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Introduction 



Enhancing the utility of evaluations is a shared goal of policymakers and 
evaluators alike (Filstead, 1980; Gideonse, 1980; Pincus, 1980; Bailey, 1979; 
Bissell, 1979; Patton, 1978; Weiss, 1977). To alter the destiny of the typical 
evaluation report from gathering dust on a bookshelf to guiding administrative 
action requires new approaches and methodologies. Although approaches for 
increasing use of evaluation results fcr decisionmaking have been documented 
(Madey, 1980; Smith, 1980; Bissell, 1979; Hayman et al., 1979; Patton, 1978), 
few formal methods exist. The need for such methods has been expressed recently 
from several perspectives (Herman and Glennan, 1980; Hill, 1980; Weiler and 
Steams, 1980). Filstead (1980), for example, argues that polic3niiakers should 
insist that procedures be employed to gather policy-relevant data from interested 
or involved parties • Policy Ii:5)lications Analysis (Stenner and Madey, 1976) 
is one such tool designed to maximize the likelihood that an evaluation report 
will have an impact on decisionmaking. PIA was designed to help people planning 
and conducting evaluations tailor their information so that it has optimal 
potential for being used and acted upon. This paper describes the development 
and application of Policy Implications Analysis (PIA). 



The paper is organized In four parts* First, the need for formal methods 
to enable decisionmakers to specify more explicitly thler information re- 
quirements is briefly discussed. Second » one such method designed to meet 
this need, PIA, is described in enough detail to permit other Investigators 
to apply the approach to their own studies. In the third section, a recently 
completed evaluation Is used to Illustrate the method's application. Asso-* 
cited advantages and disadvantages of the method are presented in the final 
section. 

Need 



The importance of designing evaluations so as to meet policymakers* informa- 
tion requirements is a current and recurring theme. Plncus (1980) describes the 
typical situation as follows: 

Most policymakers want their programs to succeed; but most "scientific" 
evaluations address effects and indicate that student outcomes as 
measured by test scores, dropout rates, and other such measures appear 
to be little affected by new government agency programs. Such reports 
of "no significant effect" are generally unaccompanied by useful recom^ 
mendations* for program improvement or policy change. Meanwhile, policy- 
makers seek to know not only about effects, but also about what is 
going on in the program: how the resources are being used, whether 
in^lemenration corresponds to program intent, and who is benefiting 
from program resource use* In effect, what can result is a "dialog of 
the deaf." in which neither party understands the other *s premises 
(pp. 3-4). 

Hill (1980) argues that evaluation planning must begin with a careful assessment 
of policymakers' information needs, beginning, in the case of a federally-mandated 
study, with the Congress. Likewise, Berryman and Glennan (1980) argue that appro- 
priate evaluation methods for a Federal program cannot be defined without 
reference to the policymaking process* Berryman and Glennan state that the 
policymaking process is political in that it involves real value conflicts; 
because no one party can impose a solution to the conflicts, policyinaking becomes 
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a mechanism for resolving differences or a process of compromise. To be most 
useful, then, evaluations should address different outcomes and processes, 
each responding to some party's interest in the policymaking process. 
Berryman and Glennan add that realities often dictate that the interests of 
the party funding the evaluation are primary. To establish agreement about the 
evaluation's goals and appropriate research designs, Weiler and Steams (1980) 
argue for increased collaboration among evaluators and government agencies. 
The goal of evaluators' work must be to provide policymakers with information 
which will help their deliberations. 

PIA was designed to improve connunication between the evaluators and policy- 
makers by providing an active forum through which information users could express 
their information needs. PIA is based on the assumption that a more responsive 
and useful evaluation will result by understanding both the policy context 
within which the evaluation is commissioned and also the questions being posed 
by actors within that context. PIA enables the evaluation team to understand 
what the policy decisions are likely to be, and to identify the types of infor- 
mation that will be needed .to make these decisions • 

Description 

PIA is a six-step process designed to explicate the information requirements 
of key information users at a variety of levels (e.g.. Federal, state, local). 
Active participation by both evaluators and policy makers is necessary through- 
out the process. The PIA method proceeds as follows: 

Step 1 . Statements of hypothetical, but theoretically possible, findings 

which could result from the evaluation are generated. The findings 
range from being very straightforward and in line with previous 
studies to being relatively unexpected (in relation to previous 
thoery and practice). Formal or informal involve!::^ nt of key in- 
formation users may be sought during this first step. 



Step 2 . Using the generated hypothetical findings, a written exercise 
is prepared for later use with a carefully selected panel of 
respondents. The exercise is comprised of four major sections: 



• an introduction to the exercise which briefly 
describes the PIA method and the program under 
scrutiny; 

• hypothetical findings or "scenarios" which might 
appear in an executive summary of a future evaluation 
report with guidelines for responding to the hypo- 
thetical findings; 

• an opportunity for respondents to generate findings which 
might result in a decision to drastically revamp or 
eliminate the program; and 



• an evaluation of the overall exercise. 



Step 3 . A carefully selected panel of respondents, representing a cross-- 
section of policymakers and other information users at a variety 
of levels (e.g.. Federal, state, local), is identified. Both 
proponents and opponents of the program are included in the 
respondent group. 

Step 4 . Each member of the respondent panel is asked to complete the 
exercise. Respondents read the hypothetical statements and 
respond to each finding in terms of the fallowing: 

• how likely he/she feels that the finding will actually 
result from the study; 

• what policy actions he/she considers feasible should 
the finding be sustained; and 

• what further inforr^ation would be needed to modify 
policy or take action based on the finding (i.e., 
what additional questions would be posed?). 

Step 5 . The responses of the individuals are analyzed and synthesized to: 

• clarify the expectations of relevant stakeholder groups 
regarding the evaluation; and 

• delineate the context within which the evaluation is 
embedded. 



Step 6 . The analyzed and synthesized responses are used to develop a set 
of policy-relevant questions or hypotheses which then guide the 
creation of a conceptual framework for the evaluation. 
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The PIA method builds upon two futures methodologies, the Delphi Technique 
and Scenario Writing. The Delphi Technique is a methodology for eliciting and 
refining ideas and gaining consensus from a panel of experts about possible 
future states or conditions. Typically, the procedure involves several "probes" 
of a panel utilizing a questionnaire, and then aggregating and feeding back the 
findings until group consensus is achieved. It should be noted that one of the 
key purposes of Delphi is not a purpose of PIA. Iterative probew are not used 
to reach a unified consensus from the panel of respondents; rather, the purpose 
is to uncover and explicate diverse expectations and information needs emanating 
from different perspectives on . ture policy and decisionmaking. However, 
several sequential PIA exercises may be utilized to increase the quantity and 
quality of design-relevant information. 

Scenario Writing, a technique perfected by Herman Kahn and popularized by 
the book. The Year 2000 ^ involves the generation of carefully calculated stories 
about the future. Scenarios have two important advantages that are relevant to 
the hypothetical findings generated in the PIA exercise: first, they call 
attention to the larger range of possibilities that must be considered; and 
second, they illustrate forcefully certain principles or questions which would 
be ignored if one insisted on taking examples only from the real— present" 
world. In the PIA exercise, the interest is on forecasting "findings" and 
generating policy "scenarios" for more than just a given time frame — for 
example, one year — in the future. In effect ^ the intent is to look ahead to 
the long-range unfolding of an e aluaticn process. Thus, in the PIA exercise, 
one must be especially alert to not being overly constrained by the routinely 
"plausible" and "conventional" in making hypothetical projections. Finally, 
in engaging in the exercise, all participants must try to experience "role 
moments" in the future. 



Application 

PIA has been used successfully by NTS Research Corporation In several 
longitudinal evaluations of Federal programs (e. g., Midey et al., 1980; McNeil 
et al., 1980). An illustration of the method's application in the recently completed 
evaluation of the State Capacity Building Program for the National Institute 
of Education, Department of Education, Is presented In this section. Prior to 
illustrating how PIA has been used In such an evaluation. It Is helpful to de- 
scribe the specific program and evaluation to be used in the example. Therefore, 
a brief overview of the State Capacity Building Program ard its evaluation is 
first presented.^ 



The State Capacity Building Program and Its Evaluation 

Through the National Institute of Education (NIE) -sponsored State Capacity 
Building Program (established in 1975 and still operating), state education 
agencies (SEAs) are awarded one-year, renewable grants of about $ 100,000 each 
to support the development, and eventual institutionalization, of statewide 
dissemination systems for making current educational knowledge and practices 
accessible to administrarors and practitioners. According to the NIE program 
announcements, such systems are to be comprised of three generic components: 
(1) an information resource base which contains the knowledge or knowledge- 
based products clients need, (2) linkages to connect the resources with the 
people who could benefit from them, and (3) a leadership/management component 
to coordinate the various activities needed so local educators could use the system 
for school improvement. From these generic components, states are expected to 
develop specific sys - :c , customized to their owr contexts, which extend or 
adapt existing structar > for enhancing dissemination services; and as a 
collaborative effort between NIE and the states, time2ines are established by 
^lutual agreement. 



Under the sponsorship of the Research and Educational Practice unit of 
MIE's Program on Dissemination and Improvement of Practice, NTS Research 
Corporation conducted a study of the first four years of operation of the 
State Capacity Building Program ~ 1975-1979. The purpose of this study was 
not to evaluate the success of specific capacity building projects, but rather 
to identify factors which facilitate or impede SEA efforts to build and in- 
stitutionalize statewide dissemination systems. The NTS study was intended 
to develop an understanding of how federal and state policy might promote 
capacity building for this program and for future capacity building programs. 
The NTS study was also intended to provide both federal and state decisionmakers 
with useful information for improving the current and future programs. 

The evaluation vas comprised of four phases: (1) a design phase 
(October 1976 - August 1977) devoted to describing the program, clarifying and 
translating the program's goals into measurable variables, and developing a 
design, appropriate instrumentation, and data collection and analysis procedures 
for the study; (2) a preparation period (September 1977 - August 1978) which in- 
cluded initial fieldwork in 23 project sites, some descriptive reporting, re- 
finements in the study design, and approval of a forms clearance package; 
(3) the full-scale evaluation (September 1978 - April 1980) which included two 
waves of quantitative data collection (Fall 1978, in 33 project states; and 
Fall 1979, in all 50 states) and an additional wave of qualitative data collec- 
tion (Winter 1980, in five project states); and C^) a dissemination phase 
(July 1980 - April 1981) in which the study's findings and implications were 
shared with policymakers, researchers, and practitioners. Figure 1 summarizes 
the four phases of the NTS study, highlights major data collection periods, 
and indicates how the NTS study fits into the overall time period of the NIE 
program. 
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Given a program so dynamic, NTS felt a need to obtain explicit informa- 
tion from the study's major audiences regarding their common and unique 
information needs and their expectations for the study. PIA was designed to 
help NTS customize the evaluation so that it had optimal potential for being 
used. First used in the study of the State Capacity Building 

Program, NTS founa that PIA was eminently suited to an evaluation of this kind, 
where program goals were innovative, where program guidelines were non- 
prescriptive to accommodate wide variatior among the participants, where project 
development was expected to be evolutionary and incremental, and where study 
audiences — and their respective information needs — could differ dramatically. 
That is, NIE program staff members, state project staff, legislators, and 
concerned personnel of state, regional and local education agencies each made 
decisions at different levels and required different information in different 
formats at different times (Madey, 1981). 

An Illustration From a Recently Completed Evaluation 

Exactly how the PIA aethod was used in the evaluation of The State Capacity 
Building Progran is explained in this section on a ste?-by-step basis. Where 
appropriate, samples from the respondents* completed exercises are included. 
The intent is to provide sufficient information for other evaluation designers 
and implementors interested in customizing the exercise for use in other 
program evaluations. 

Step 1 . Statements of hypothetical, but theoretically possible, findings 
which could result from the evaluation were generated . The 
findings ranged from being very straightforward to being 
relati"/ely unexpected . Informal involvement of key information 
users was sought . 



Findings were generated for each of the three generic component areas 
included in the program. A total of 29 finding statements were gen- 
erated, 13 for information resources, 8 for linkages, and 8 for 
leadership/management. The NTS project team was assisted by several 
state project directors, but federal program personnel were not in- 
volved in generating finding statements. (Subsequent exercises -included 
such federal personnel, as well.) Sample findings for each program 
cou^onent are included in Figures 2-4. 

Step 2 . Using the generated hypothetical findings, a written exercise 
was prepared . 

An annotated outline of the four-part exercise is presented 
in Figure 5. 

Step 3 . A carefully selected panel of respondents, representing a cross- 
section of policymakers and other information users at a variety 
of levels, was identified . 

Respondents included federal program managers, federal project 

monitors, state project directors and recognized experts in 

the field. Respondents were selected and invited to participate 

in the PIA exercise. 

Step A . Each mmber of the respondent panel was asked to complete the 
exercise. Respondents read thB h^-pothetical findings and 
responded to the included guidelines and probes . 

All invited panel members completed the exercise. Sample responses 
to the three hypothetical finddLng statements are also included in 
Figures 2-4. 

Step 5 . The responses of the individuals were analyzed and synthesized 
to clarify the expectations of relevant stakeholder groups and 
to delineate the context within which the evaluation was embedded . 

A synthesis report was prepared and the results used in finalizing 

the evaluation design. Perhaps, most importantly, PIA revealed that 

the federal program staff had a greater interest in understanding 

program design and management factors than had been originally 

O thought. PIA also revealed that although evaluation information 

ERIC ^ 



would be useful at the state level, the primary clients were 
personnel at the federal level. Many state projects would be com- 
pleted before the final evaluation report was published and disseminated. 

Step 6 . The analyzed and synthesized responses were used to develop a 

set of policy-relevant questions or hypotheses whic : then guided 
the creation of a conceptual framework for the evaluation . 

Policy-relevant questions were developed for the overall program 

and major program components. The two initial design questions 

were as follows: 

• Is capacity being built as a result of this program? 

• Is the program having an effect? If so, what is the 
nature of the effect? 

The PIA exercise provided information which enabled the NTS evaluators 

to refine the major study questions and final study design. Actual 

findings were summarized under three major research questions: 

• Is dissemination capacity being built? 

• What are the factors affecting the building of capacity? 
What factors help or hinder achievement of program 
objectives? 

• What program management and program design factors 
affect the building of capacity? 

Selected findings and associated policy implications from the final 
report (Madey et al., 1980) are excerpted in Figures 6-9, Findings 
for the three major program components (i.e., information resources, 
linkages, leadership/management) are included* In comparing the hjrpo- 
thetical findings presented in Figures 2-4 with the actual findings 
presented in Figures 6-9, it must be emphasized that almost four 
years passed between the time cf the initial PIA exercise and genera- 
tion of the final report. 



InforMtloQ Resources 

A- BTpocheClftAl Fisdlag St^cefflenc 

Although all staces subscribe Co the notion that dissemination involves 
two-way co maunl cation and that the inforaation base should evolve in re- 
sponse to user deaand» it vas found that in twenty of the states the 
nature of the infonaation resource base was largely detcralncd by SEA 
staff wich little foraal or informal assessoent of user needs. Project 
staff said that political and economic exigencies dictated the scope of 
the information resource base. 

6. Guidelines (Circle your response) 

1. To what degree is this finding within the purview of the NTS , 
evaluation? 

Definitely within Definitely without 

© 2 3 A 5 6 7 8 9 10 

2. How much knowledge do you have of the general area addressed by 
this finding? 

Koch knowledge Little knowledge 

^ 2 (T^ 4 5 6 7 8 9 10 

3. To what degree does this finding correspond wich your expectations? 

Recced jlot ejected 

12345(7)789 10 

4. To what degree does this findir^ have lanedlace policy implications? 

To a large extent To a limited extent 

1(2)3 4 5 6 7 8 9 10 

5. Given chat this finding accurately reflects reality, is ic scaced la 
a concise fashion, i.e., does it cocaunicat^? 

CocBcinicaces well Communicates poorly 

12 ^ © ^ '^'^ ^ 3 9 10 

C. For a Finding Selected for Significant Policy Implications 

1. "Sihat policy acclon<s) might be precipitated by this finding? 
^ "B^iild needs assessment component into project-** 

2. Hhat further information would you need to modify policy or take 
action based on this finding? (What additional questions vould you 
pose?) 

"Validity of 'political/economic constraints' pleadings." 

"Heconaendations on how to perform useful needs assessaenc." 

**Becoc3Dencations on how SZAs could acquire greater credibility 
with teachers and LEA adainistrators." 

D. Finally, for the Information S.esources Coaponent: 

Vrite la che space ielsw a "finding atacca.int" in this cccponent chat 
would supporc a reconacndcrion on your part to drastically revamp the State 
Capacity Suildiag Program. 

"Document^ derscs^jz by user's that they are aeek-ing very different znfar^ 
raticn fron vkct they're ge-stinc. For e:~Dle, -in cn SEA ^csserticZln 'ua-Lrjs 
ERIC cs its resource base, dersard by teachers (say u^t a level of cClf or 
ncre of all teacher iKcurlries) that they get info r raticTz on and actual 
acafzples of instructicrjal rxizerials Ce.g.^ 2nd grade arithmetic series, 7th 
grade social studies for Chicago szudents) 

FIGURE 2. An Illustration of the PIA Exercise for the Infonaation Resources 
Prograo Coaponent 



15 



Linkages 



A. Hypothedeal Finding Seacemeac 

Oaly five of che twenty-fcur states employ full- 'rise linking personnel funded 
through their own budget (or through budgets of latcrsicdiace units or LEAs). 
Full-time linking agents appear to provide better services than do those who 
perform a linkage function in addition tc other functions. Most states use a 
combination of full-and part-tine linkers. Part-cisre linkers ser^/e aiore as a 
funnel for (pre-packaged) information packages prepared vithin the resources 
component; full-time linkers perform transformations of information and work 
closely vlth the clients. 

B. Guidelines (circle your response) 

1. To what degree is this finding within the purview of the XTS 
evaluation? 

Definitely within Definitely without 

1 (2) 3 4 5 6 7 8 9 ' 10 

2. How much knowledge do you have of the general area addressed 
by this finding? 

Much knowledge tittle knowledge 

123(1)56789 10 

3. To what degree does this finding correspond with your expectations? 
Expected Hot expected 
12(3)456 "7 89 10 

4. To what degree does this finding have Immediate policy Implications? 
To a large extent To a limited extent 
12(T)45678 9 1C 

5. Given that this finding accurately reflects reality, is it stated 
in a concise and clear fashion, i.e., does it communicate? 

Cocaainicates well Communicates poorly 

1 2 3 4 5 (T) 7 8 9 10 

C. For a Finding Selected for Significant ?olicy Implications 

1. *Jhat policy actlon(s) might be precipitated by this finding? 

"The policy implication is clear — hire more full-tine agents — but 
it does not take account of Resource constraints." 

What further information would you need to modify policy or take 
action based on this finding? (What additional questions would 
you pose?) 

••Information not specific enough to draw conclusions. Costs? 
In ^t way are the services provided 'better'? What 'better* 
results are obtained by working more closely with clients? Are 
the extra costs worth it?" 

D. Finally, for che Linkage Component: 

Write in the space below a "finding statement" in this component that 
would support a recommendation on your part to drastically revamp the Stare 
Capacity Building Program, 

''Veers prefer dcczcr^er^t^caaed iKf7rrx:tion system at their i^zr^ediate 
ccmsnd md^dhere zhey k^e it^-^firid -iz of freezer utility . " 

''This finding' i)ould be cut of keeping^ -Jith the bas-ic prerrise of the 
^ro^rm thst it (the Frogrsrt) zmcld hcrje to be changed rsdicaZZy, " 

FIGURE 3. An Illustration of the ?1A Exercise for the Linkage Progran Cocponent 
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Zn «lcv«n of zhm cweacy-four scac«s, ch« Sc*c« CAp*clcy Building Craac 
•ccouacs for l«ss chan 10 percent of the cocal SEA expeodlcurcs for disseo-^ 
Inadoa. la chcse scaces the lapacc of no Scace Capacity Biillding Crane 
would be largely incoasequcocial Izi cems of either che large misber cf 
sexvlcea available la che scace or che quality of the delivery aechaalsas. 
Iheae sasae states have key le»<i«rs funded outside the grant who are well 
eacrenched la che power structure and who appear to be developing statevlde 
capacity largely independent of che Scace Capacity SuUding Crane. 

B. Goldellaes Cclrcle your response) 

1. To what degree is this finding vichin the purview of the HIS 
evaluation? 

Definitely within Definitely wlchouc 

1^3456789 10 

2« Bov ouch kaovledge do yoa have of che general area addressed by 
chls finding? 

Much kaovledge Llccle knowledge 

12(7)4 5 6 7 8 9 10 

3. To whac degree does chls finding correspond vlch our cxpectaclons? 
«?«cced Not expected 

1 2 3 4 5 6 (T) 8 9 10 

4« To vbat degree does this finding have Inaedlate policy iapllcatlona? 

To a large esrenC To a Halted extent 

^ 23456789 10 

5. Given that chls finding accurately reflects reality, is ic staccd in 
a concise and clear fashion, i.e. does ic coasxunlcace? 

Cooeunlcaces vdl Cooauaicaces poorly 

1(2)345673910 

C. For a Finding Selected for Sigalf leant Policy Ixplicacions 

1. Whac policy accioa(s) alghc be precipitated by chls finding? 

a. "Give no awards co the 'have* states. This is probably 

not a polldcaliy feasible option, however." 

b. '^Tbc prograa should be differentiated Inco two pro grass, 

l.e- : 

CD. A. canaclry building grant prograa for states with 
iw ik to low dissealnatlcn systess already is 
operation; and 

C2}. A supplesMntary dissealnatlcn grants prograa In 
states vlth aore mature* established systems to 
supporr demonstration* Cfor other states), 
dlsseslnatlon across states, experlaentation, 
and evaluation." 

2. What further Inforaation would you need to aodify policy or take 
action based on this finding? (What additional quesdons would 
you pose?) 

Inforxation chat would allow developoent of an activity 
appropriate to the^ eleven states to further their disseainatlon 
activities* since they don't need capacity building auards. 

D. Finally, for the Leadership/Manageaent Cooponent: 

tfrite in the space below a "finding stateaent** in this component 
chac voulc supporc a recozaendatlon- on your aprc to drascically revaap 
che Scace Capacity Building Prograa. 

"^c^ t^r Qc^^vi.t'i^s riaz out of car. Inz^smsdrlst^ Servisa 
Acgrcj (ISA) CT zecertzml-izid •ir.forrizticn cffz^9 are ncr^e 
susceaefut zhm. zh^se aer . t z nlCzed -in zrd rur. 3^ c Stzta 



Ficnsz 4. An Illustration of the ?IA Exercise for the Leadership/ 
Management Prograa Cooponent 



I. Introduction 

A brief description of the PIA method and the program to be 
evaluated* CSee pp»l-9 for an illustration) ♦ 

II. Hypothetical Finding Statements CSee Figures 2-4 for illustration). 

A. "Scenarios" which might appear in the executive suimnary of a future 
evaluation report on the program 

B. Guidelines for responding to each hypothetical finding or scenario: 

1. To what degree is this finding within the purview of the study? 

2. How much knowledge dc you have of the general area addressed 
by this finding? 

3. To what degree does this finding correspond with your 
expectat ions? 

4* To what degree does this finding have immediate policy implications? 

5, Given that this finding accurately reflects reality, is it 
stated in a concise and clear fashion, i.e., does it 
communicate? 

C. Select findings with greatest policy implications, and determine: 

1. What policy action(s) might be precipitated by this finding? 

2. What further information would you need to modify policy or 
take action based on this finding? (What additional questions 
would you pose?) 

III. Prepare a "Finding Statement" that would support a recommendation on your 
part to drastically revamp or eliminate the program. (See Figures 2-4) * 

IV. Critique the Overall Exercise: (See Attachment for actual critiques.) 

1. What is your overall reaction to this PIA process? 

2. Did it afford you a vehicle for persenting your t-houghts 
about the evaluation and its potential contributions? 

3. What modifications to Policy Implications Analysis wuld 
you suggest to improve the procedure? 



FIGURE 5. An Annotated Outline of the PIA Exercise 



Is Capacity 3elo^ Butit? 

!• Th€ prloary effect sought frog che pro^raai-^lncreasgd cacaclty 
of SEAs for dlssealaatlon-*ls being, achieved . 



(Inf onaaclon • Scaces have substandally Increased che breadch and variety of 
Resources) knowledge resource bases that caa be accessed through the S£A 

disseoination unit. ^ 

(Linkages) • States have oodlfled existing structural arrangcoeats to develop 
Che capacity for the delivery of inforxation to clients through 
"linkers*' who function as inforoation brokers. 

(Leadership/ • Coordination of, and cooperation between* various pro^aa and 
Xanageoent) service units in the SEA and in the state has been Inproved 

leading to a acre coaprehensive and general izable body of resources 
available for dissesiination to the education system* 

( Le adership/ • !2osc states Izi th< SC3P evidence 20vi±aaent toward iasticuclonalizing 
Manageaaent) their diss ea i n ation capacity, although it is still too soon la that 

process to deterziine If the disseaination systea will indeed be-- 
cooe an accepted part of SZA program services offerdLngs. 



2. The -process of Increasing canacity follows several different patterns 
depeadlng on state history and context^ and reflects the flexibility 
allowed by the pro^aa guidelines . 

(Info raat Ion • Resource base development has expanded prlaarily in the areas of 
Resources) proaising practices and other state and local Inforaatien files. 

It appears that in oost states reliance Is placed upon validated 
programs in the school iizprovement process; less esphasis is placed 
upon Information gained from non-validated, promising practices 
as a basis for school Izprovecent. 

(Linkages) • For the delivery of services, three linkage patterns — which we 
have characterized as SZA controlled (tightly coupled), SZA 
coordinated (loosely coupled), and external (uncoupled) — appear 
to reflect state philosophy and consequent structures for school 
improvement. 

(Leadership/ • Coordination has been Improved primarily between the capacity 
^o ageaent) building projects and generic programs such as l^HS and Title 17; 

less coordination has been achieved between the project and content 
specific programs, such as vocational education and handicapped 
education. 

(Leadership/ • Building SEA dissemination system capacity seems to have an 
Management) identifiable sequence of development, but individual state factors, 

and c h a ng es in those factors may override this "developmental" pattern. 

FIGURE 6, Summary of Findings for First Major Research Question, 
••is Capacity Being 3uilt?" 



Source: Madey et al.. Building Capacity for the Imoroveaent of 

Educational Practice: An Evaluation of :^IE's State Dissea- 
ication Grants Program, Volume I: Final Evaluation Report . 
Report prepared for the N"ational Institute ef Education under 
Contract #400-76-0166. Durham, :i.C.: NTS Research Corpora- 
CloQ, 1980, pp. 7. 10-7. 12. 
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Factors Affecting Program Success 



Success of SEA efforts to implement and institutionalize dissemination 
systems appears to be influenced by the following: 

State Factors 

• Continuity of energetic and entrepreneurial leadership; but once 
that leadership is gone the process may become endangered. 

• Previous involvement in dissemination activities is a helpful 
but not sufficient factor in institutionalization. 

• Placement in an administrative unit appears to assist in the 
development of coordination and comprehensiveness of the system. 
Placement in a service unit appears to assist in the delivery of 
services to clients and the institutionalization of the system 
in the SEA* 

• Initial strategies of targeting clients for service and developing 
products for use by particular clientele enhance the development 

of coordination and comprehensiveness of the system. But the project 
needs to move on to serve the general clientele if institutionalization 
is to be enhanced. 

• The active support of SEA administrators (Chief State School Officers 
and their associates) is crucial to building capacity and implementing 
and institutionalizing the dissemination system. 



Other Structural Factors 

• The continued fragmentation of the dissemination components of 
Federal programs impedes the building of generalized and 
comprehensive dissemination systems within the states. 
Despite the fragmentation, however, many states have made 
progress in coordinating dissemination efforts at the state 
level . 



FIGURE 7. Summary of Findings for Second Major Research Question, 

"What are the Factors Affecting the Building of Capacity?" 

Q 'Jource: JIadey et al.. Ibid., pp. 7.12-7.13. 

ERIC 



Program Design and Management Factors Affecting Program Success 



Success of SEA efforts to implement and institutionalize dissemination 
systems appears to be influenced by the following program design and 
management factors: 

• Collaborative planning and flexibility of Program guidelines 
permitted states to tailor their dissemination projects to fit 
their individual contexts. While these approaches have 
enhanced the in-state capacity for independent solutions to 
dissemination system development, they may also foster 

areas of non-clarity of purpose between NIE and the states. 

• Opportunities to communicate with personnel from other 
states and agencies facilitate project development. Although 
the Program provided mechanisms for such communication .and 
for technical assistance, these provisions appear to be too 
limited. In other words, the plan was appropriate; its 
implementation was not adequate to meet the needs of the states. 

• Program objectives regarding the role of the dissemination 
system in relation to a state's other school improvement 
efforts are not adequately specified in program guidelines 
and project proposals. The result is that the potential for 
facilitating the use of new knowledge and educational 
practices for school improvement and equal educational 
opportunity is only partially seen and realized in many 
states participating in the Program. 

• Program and project goals for increasing equity and for 
operationalizing those goals are not well developed. There 
is little evidence of program resources being directed ex- 
plicitly and in concerted ways for increasing equity in 
education. 

• NIE staff resources assigned to this Program have been 
too limited to provide the necessary monitoring and 
technical assistance needed and often requested by the 
state projects. 



FIGURE 8. Summary of Findings for Third Major Research Question, 
"What Program Design and Management Factors Affect the 
Building of Capacity?" 



Source: Madey et al.. Ibid. , p. 7.12-7.13, 
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Sunpary of Policy Ipplicatlons 

Th« following policy tapllcacloM arc presented within the context of 
change and uncertainties at tha Tederal and state levels. These uncertainties 
are reflected In questions abcuc expanded Federal leadership of the growth and 
application of dissemination systeos for assisting in attaining educational 
inprovement and education equity. At the state level, there are questions of 
increasing pressures on budgets for educational activities and of the willing* 
ncss of states to commit themselves to continue and refine -.e use of dissemina- 
tion systems, for educational improvement and educational equity. 

1- Collabora tively Strengthen ?rograa Conceptualizacion and Design 

The findings of this study have broad implications for future prcgrams. 

but in the near-term. N1E and the states should work together to 

strengthen the Program t'orough: 

• A clearer conceptualization of. aod specification or the guide- 
lines for. ways states can uce dissemination resources to facilitate 
significant improvements in educational practice and cqxiity — 

e.g.. in connection with other SEA programs or through other external 
linkages with practitioners. 

• A clarification of the priorities or guidelines for types of re- 
sources that should be further deve::oped — e.g.. those that are 
most used* most useful, cost dixflcult to obtain through other 
means, or most relevant to equity Issues in education. 

• The provision of needed linker training, particularly to enhance 
skills of indlvld»ials who are already located in positions to 
facilitate school improvement. 

2. Strengthen Program ?*anagesent and LeAdershio 

• NI£ staff resources for this Program should be sl.-engthened in crder 
to provide more guidance on critical project Issues — e.g.. utilisa- 
tion of dissemination to enhance equality of educational opportunity, 
and trade-offs among alternative ways the states are authorized to 
use the Program resources. 

• Onycing and viable communication mechanisms among the states involved 
in building dissemination capacity should be created and maintained, 
ThAse mechanisms could include the regional exchanges who could 
function as the vehicle through which comunication among states 
idthin regions is maintained. 

• Guidelines should acknowledge the development or organizational 
capacities and provide assistance for critical functions at each 
stage. A "step-wise" or "building block" approach is recommended 
that is keyed to three stages — planning, implementation and 
ins ti tu t ionaliza tion . 

3. Improve Federal Level Coordination Mechanisms 

• Sfec h a nism s for improving coordination of (or support for the cooperation 
of) Federally-funded programs should be created at the Federal level, 

^- Examine Further the Secondary or '^Downstream" Effects of the Program 
In Terms of Its Effects on Education 

• This study shows that capacity is being built, and identifies 

a number of factors that are enhancing and limiting the capacity 
building effort. The Program should be examined further to de- 
termine how the capacity is used and what apsects of dissemination 
capacity are most critical in achieving improvements in equity and 
practice in education. 



FICCRE 9. Summary of Policy Implications for the JTTS Study of 
The State Capacity 3ulldlng Program 



Source: 




Associated Disadvantages and Advantages 

Enumerating the pros and cons of any method is a useful exercise. The 
potential disadvantages and advantages associated with FIA are summarized 
in this section. In addition, respondent critiques of the first overall 
exercise are presented in the attachment at the conclusion of this article. 
(Modifications recommended by the first respondents were incorporated into 
the procedures described in this article.) 

Potential Disadvantages 

Perhaps no technique is without potential disadvantages and PIA is no ex^ 
ception* Some of the difficulties surround the implementation of the technique 
and are presumably correctible through refinements in the process. Other 
problems stem from the inherently obtrusive character of the technique itself* 

The technique takes time, a commodity that is often rare in the initial 
stages of an evalxiatioa. If a single polling of respondents is all that is 
desired, then the entire process of scenario design, administr^.tion, and 
analysis can be accomplished in six weeks. If, however, it is important to 
obtain consensus on the information priorities, then several iterations may be 
desirable, each requiring, at a minimum, one month. 

Selection of the panel can be problematical. Diversity of position and 
perstiasion is essential but either too much diversity, or diversity at the 
expense of representativeness, can be self-def eating • The ideal is to ade^ 
quately represent the major constituencies that will finally use the evalua- 
tion information. In some evaluations, there may be only one user group; wher^s 
in others, such as the evaluations of Head Start and the State Capacity Buildijig 
Program, the range might run from local program staff to Congress. It should 
be apparent that an inappropriate panel selection will hamper the generalizabi^ity, 
and thus, utility of panel responses. 
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Related to the issue of panel selection is the fact that policy contexts 
are dynamic and the important actors and/or information users may change with 
time* A partial solution to this problem is an annual polling of users 
accompanied by a reassessment of each user^s continuing relevance. Of course, 
this solution is less effective in highly dynamic contexts in which the 
significant actrrs, or political climate, frequently change. 

Lastly, Policy Implications Analysis is obtrusive and may reawaken dormant 
policy issues which are best left alone. The technique may promote frustra- 
tion by encouraging participants to explore policy actions and alternatives 
for which adequate information does not exist Cat the time) to reach an informed 
decision. Depending upon the policy context, the energy generated as a by- 
product of the technique may be viewed as desirable or undesirable • 

Selected Advantages 

Even with such disadvantages, PIA represents a methodological advancement 
for policy research and evalaution; inherent in the technique are substantial 
advantages: 

• Policymakers' and other information users ^ expectations 
and preconceptions regarding the evaluation and its 
findings are made explicit. 

• Policy alternatives are delineated and supporting 
information requirements for each alternative are 
identified (i.e., evaluation questions are formulated). 

• The connections between evaluation information and 
alternative policy actions are given additional clarity. 

• Areas of consensus and dissensus among information users 
are identified. For example, information needs of program 
staff and higher level policymakers are not always congruent, 
and thus, perceptions of the purpose and benefit of the 
evaluation may differ. 

^ Possible unintended outcomes are unmasked by involving 
information users who are not totally supportive of the 
program. Sjiich outcomes generally go unaddressed when inten- 
tions and program design comprise the sole foundation upon 
which the evaluation is built. 
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• Boundaries for the evaluation, and information priorities 
vithln these boundaries, are made explicit. 

• Results of the PIA exercise often result in deflating un- 
realistic expectations regarding what can be learned from 
an evaluation. 

• A large number of respondents may be involved in the process 
without restrictions inqjosed by geography. The process is 
relatively lov in cost, compared to the benefits derived 
from using such a tool. 

• If desired, the entire process can be accomplished xrLth 
anonymity for the participants, thus avoiding unnecessary 
ideological battles and policy confrontations which might 
better await the arrival of objective evalxiation information. 

Most importantly, though, PIA has already been used successfully to enhance the 

utility of evaluation results for decisionmakers. This formal method fills a 

critical need, and NTS Research Corporation's experiences with PIA suggest that 

it may be useful to others as well. The response to the statement "evaluations 

aren't useful" should no longer be tacit agreement. PIA can be used to help 

evaluators design and implement studies which meet policymakers' needs. 
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Attachment. RESPONDENT CRITIQUES OF THE FIRST PIA EXERCISE 



(1) WHAT IS YOtm OVERALL REACTION 70 THIS PIA PROCESS? 
Federal Respondent: 

• I found this to be an enlightening task. Thanks. After 
going through your mock findings I tried to think of what 
might be missing. Ideally, I think findings should be 
presented in 3 ways C3 kinds of policy implications): 

1. How can states better manage their projects? 

2. What new projects should ;iIZ initiate to alleviate 
weaknesses? 



3. Did the State Capacity Building Program do any good? 

yty consistent problem was getting enough specific information from 
the mock findings to judge the policy relevance. Probably a weakness 
in this tool we must live with. Hope you continue to use the method. 

Federal Respondent: 



Idea is excellent. This particular form of the idea is 
useful but limited. In its present form, it seems to have 
more to say to SZA's than to the Feds. G*'ill Final Report 
be "modularized" or will both States and Feds receive 
exactly the same report?) 



Federal Respondent: 

• I can see that it would be useful to JJTS in helping dec'-'o 

what data to collect. But the approach concentrates on micro 
rather than macro questions so that bigger issues don't get 
surfaced . 



Federal Respondent: 
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I thiah the finding statements are probably geared veil for 
state purposes - managing program - making shifts based on 
one thing or another. I have a problem with Federal implications. 
We're going to have Cwith ESZA renewal in 78) to deal with 
questions of coordinated Federal policy in dissemination and 
how to accomplish that.... are present efforts fragmented - 
too much of a burden for states to try to coordinate? What 
• are the future cost options Cwhat about means to continue 
activities after Federal funds cease)? We have some other 
options too - say if full-time personalized linkage is found 
to be overwhelmingly necessary and useful > we can emphasize 
that in programs; or if currictilum materials are 85Z of the 
user requests, we have to revamp the assumption about infor- 
mation needs; or if users of service are all located within 
15 miles of resource centers, we can urge decentralization. 

- 2 b 



Federal Respondent: 

• Evocative, enjoyable, probably informative to NTS and NIE 
but too time-consuming to attain wide-ranging responses. 

State Respondent: 

• I found it interesting and useful. Too much time, however, 
whic h I had not counted on. 

State Respondent: 

• The process may be workable, but it needs some modifications. 
Advisory Council Respondent: 

• Excellent exercise... I really got into this experience and 
thought it useful for me. For what it's worth, I spent five 
hours in reading and thoughtfully responding to the PIA. 

(2) DID IT AFFORD YOU A VEHICLE FOR PRESENTING YOUR THOUGHTS ABOUT THE 
EVALUATION AND ITS POTENTIAL CONTRIBUTIONS? 

Federal Respondent: 

• Not adequately; however, this is better than nothing, 
which is probably xahat NTS would have got from me without 
this exercise. 

State Respondent: 

• Yes, almost completely. 
Advisory Council: ^ 

• Yes, hypothetical findings showed me really great insight 
into (my) issues in the SEA dissemination system. 

(3) WHAT MODIFICATIONS TO POLICY IMPLICATIONS ANALYSIS WOULD YOU SUGGEST 
TO IMPROVE THE PROCEDURE? 

Federal Respondent: 

• For the most part, the questions address management alterations 
in the program. This is like generals fighting the last war - 
by the time the resxilts are in, most of the initiating awards 

, will have been made. Of most interest to me is how we can 
build in continuation after the Federal bucks cease to flow 
and what we can learn to apply to other dissemination efforts. 



• Talk to NIE staff before developing findings. 



• Have a brief description of the program in introduction. 
Hard for a reader unfamiliar with project to know what's 
being discussed « 

• Use conference techniques for getting at applications; 
policy shifts, additional information required. 

State Respondent: 

• I'm afraid I reacted sometimes as the project director in 
one state and sometime in light of the overall impact in all 
capacity building states, I'm not svre whether it would be 
significant to pull out these differences, I also reacted 
poorly to the final statement in each section about "dras- 
tically revamping/' Isn't evaluation info useful if it only 
results in modest modifications? Surely everything we're 
doing isn't wrong I 

• Needs to be shortened. The time required to respond is 
grudgingly given. 

• A rating scale with 10 choices violates what I have been 
taught about item construction. What will you conclude 
from an item with an average rating of 3 or 8? Five 
choices would have been more than adequate. 

• I don't understand the purpose of guidelines i?l and #5. 
They should be dichotomous choices in my opinion. 

Advisory Council: 

• Reduce the number of gradations on multiple choice (couldn't 
differentiate between 8 and 7, etc.) 

• Eliminate thfe request Cat least for Advisory Council types) 
to have to suggest a new finding. I found it very difficult. 
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NOTES 



1. Paper presented at the Ath Annual Meeting of the Evaluation Research 
Society, Washington, D,C., November .19-21, 1980, The NTS evaluation of the 
State Capacity Building Program is supported with federal funds from the National 
Institute of Education, Department of Education; however, the contents of this 
article do not necessarily reflect the views or policies of the National Institute 
of Education, or the Department of Education, Copies of the paper may be 
obtained by writing the authors at the following address: NTS Research Corporation, 
2634 Chapel Hill Boulevard, Durham, N.C, 27707, 

2, For a more extensive description of the NIE-sponsored State Capacity 
Btailding Program and its evaluation, readers are referred to the five volumes 
prepared by NTS Research Corporation which comprise the final report. The 
complete set of volumes, each of which has the same general title. Building 
Capacity for the Improvement of Educational Practice , is as follows: 

Volume I: An Evaluation of NIE's State Dissemination 
Grants Program: Final Evaluation Report 
(December 1980) 

Voliame II: 1979 State Abstracts: State Dissemination Efforts 
(October 1980) 

Volume III: A Study of Linker Agent Activities and Roles 
(October 1980) 

Volume IV: A Study of The Development of Scales Measuring 
Dissemination Capacity 
(December 1980) 

Volxame V: An Evaluation of NIE's State Dissemination 
Grants Program: Executive Summary 
(December 1980) 
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